Image processing apparatus, alignment method and storage medium

ABSTRACT

An image processing apparatus includes first alignment means configured to perform an alignment in a horizontal direction on a plurality of two-dimensional tomographic images based on measurement light controlled to scan an identical position of an eye according to a first method, and second alignment means configured to perform an alignment in a depth direction on the plurality of two-dimensional tomographic images according to a second method that is different from the first method.

BACKGROUND OF THE INVENTION Field of the Invention

The disclosed technology relates to an image processing apparatus, analignment method and a storage medium.

Description of the Related Art

An averaged image or an OCTA (OCT Angiography) image can be generatedfrom a plurality of two-dimensional tomographic images acquired byoptical coherence tomography (OCT). Because a target's eye moves, analignment of a plurality of two-dimensional tomographic images may berequired for generation of an positional difference amount.

Some of alignment methods applying an image correlation betweentomographic images may require more calculation amount and may beperformed at a lower speed. European Patent Application Publication No.2693397 discloses performing an alignment based on a boundary line of aretina layer detected from a tomographic image for high-speedtomographic image alignment.

SUMMARY OF THE INVENTION

An image processing apparatus according to an aspect of the presentdisclosure includes a first alignment unit configured to perform analignment in a horizontal direction on a plurality of two-dimensionaltomographic images according to a first method, the plurality oftwo-dimensional tomographic images being obtained based on measurementlight controlled to scan an identical position of an eye, and a secondalignment unit configured to perform an alignment in a depth directionon the plurality of two-dimensional tomographic image according to asecond method that is different from the first method.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a configuration of an image processingsystem.

FIGS. 2A to 2C are diagrams for explaining a structure of an eye and atomographic image and a fundus image thereof.

FIGS. 3A and 3B are flowcharts illustrating examples of flows ofprocesses in the image processing system.

FIG. 4 is a flowchart illustrating an example of a flow of a globalalignment process.

FIG. 5 is a flowchart illustrating an example of a flow of a localalignment process.

FIGS. 6A to 6C are diagrams for explaining examples of a projectedimage.

FIGS. 7A and 7B are diagrams for explaining an example of a lineprojected image alignment.

FIGS. 8A to 8C are diagrams for explaining an example of a boundary linealignment.

FIGS. 9A and 9B are diagrams for explaining an example of a referenceimage selection.

FIGS. 10A to 10C are diagrams for explaining alignment and imageselection.

FIGS. 11A and 11B are diagrams for explaining an example of imagedeformation by global alignment.

FIG. 12 illustrates diagrams for explaining an example of localalignment.

FIGS. 13A and 13B are diagrams for explaining an example of localalignment.

FIGS. 14A to 14D are diagrams for explaining an example of globalalignment.

DESCRIPTION OF THE EMBODIMENTS

Such a conventional method does not consider a feature (such as aposition or a disease of a blood vessel) of an inner retina because analignment is performed based on a boundary line of a retina layer thatis information regarding a retina external form. This may leave apositional deviation of a tomographic image even as a result of anaccurate alignment based on a boundary line.

The embodiments described below can achieve both improved speed andaccuracy of an alignment of tomographic images. Each of the embodimentsof the present invention described below can be implemented solely or asa combination of a plurality of the embodiments or features thereofwhere necessary or where the combination of elements or features fromindividual embodiments in a single embodiment is beneficial.

Embodiment 1

With reference to the drawings, a first embodiment of the presentdisclosure will be described below. It should be noted that an imageprocessing apparatus according to Embodiment 1 may generatetwo-dimensional tomographic images with reduced noise by performingalignment of a plurality of tomographic images quickly and accurately,and proper selection of an alignment reference tomographic image foraveraging. Numerical values according to the following embodiments aremerely given for illustration purposes, and the present disclosure isnot limited by the disclosed numerical values.

According to this embodiment, even in a case where a retina layerdeforms due to an involuntary eye movements during fixation within atwo-dimensional tomographic image, a high quality two-dimensionaltomographic image can be acquired. Here, the expression “high qualityimage” refers to an image with a higher S/N ratio than that resultingfrom one imaging operation. Alternatively, the expression “high qualityimage” may be an image with an increased amount of information fordiagnosis.

An image processing system including the image processing apparatusaccording to this embodiment will be described in detail.

FIG. 1 illustrates a configuration of an image processing system 100including an image processing apparatus 300 according to thisembodiment. As illustrated in FIG. 1, the image processing system 100 isimplemented by connecting the image processing apparatus 300 to atomographic imaging apparatus (also called an optical coherencetomography or OCT) 200, a fundus imaging apparatus 400, an externalstorage unit 500, a display unit 600, and an input unit 700 via aninterface.

The tomographic imaging apparatus 200 is an apparatus configured tocapture a tomographic image of an eye. An apparatus to be used as thetomographic imaging apparatus may include an SD-OCT or an SS-OCT, forexample. Because the tomographic imaging apparatus 200 is a knownapparatus, any repetitive detail descriptions will be omitted. Imagingof a tomographic image thereby in response to an instruction from theimage processing apparatus 300 will be described.

Referring to FIG. 1, a galvanometer mirror 201 is configured to scanmeasurement light in the fundus and defines an imaging range for thefundus by the OCT. A drive control unit 202 is configured to control adriving range and a speed of the galvanometer mirror 201 to define animaging range and a number of scanning lines in a planar direction(scanning speed in the planar direction) in the fundus. Although thegalvanometer mirror is illustrated as one unit for simplicity, thegalvanometer mirror in reality includes two mirrors for X scanning and Yscanning and can scan a desired range on the fundus with measurementlight.

A focus 203 is configured to bring the retina layer of the fundus intofocus through an anterior eye portion of a target eye. The measurementlight is brought into focus to the retina layer of the fundus throughthe anterior eye portion of the target eye via a focus lens, notillustrated. The measurement light applied to the fundus is reflectedand scattered by the retina layer.

An internal fixation lamp 204 includes a display unit 241 and a lens242. The display unit 241 may be a plurality of light emitting diodes(LEDs) arranged in a matrix pattern. The lighting positions of the lightemitting diodes are changed in accordance with an area to be imagedunder control of the drive control unit 202. The light from the displayunit 241 is guided to the target's eye through the lens 242. The lightemitted from the display unit 241 is 520 nm long, and the drive controlunit 202 displays a desired pattern.

A coherence gate stage 205 is controlled by the drive control unit 202for addressing a difference in eye axial length of the target's eye, forexample. The term “coherence gate” refers to a position at equal opticaldistance of measurement light and reference light in the OCT. Theposition of the coherence gate may be controlled according to an imagecapturing method for imaging on the retina layer side or a deeper partof the retina layer. Next, a structure and images of an eye to beacquired by the image processing system will be described with referenceto FIGS. 2A to 2C.

FIG. 2A illustrates a schematic diagram of the eyeball. FIG. 2Aillustrates a cornea C, a crystalline lens CL, a vitreous body V, amacular area (having the fovea at the center of the macula) M, and anoptic disk D. The tomographic imaging apparatus 200 according to thisembodiment will be described mainly in a case where a posterior fundusretina is to be imaged which includes the vitreous body, the maculararea, and the optic disk. The tomographic imaging apparatus 200 canimage an anterior eye portion of the cornea and crystalline lens, thoughit is not described herein.

FIG. 2B illustrates an example of a tomographic image of the retinaacquired by the tomographic imaging apparatus 200. FIG. 2B illustrates aunit AS for image capturing based on an A-scan of an OCT tomographicimage. A plurality of A-scans configures one B-scan (in other words oneB-scan includes a plurality of A-scans). The B-scan is called atomographic image. FIG. 2B illustrates a vitreous body V, a macular areaM, and an optic disk D. FIG. 2B further illustrates a boundary L1between an inner limiting membrane (ILM) and a nerve fiber layer (NFL),a boundary L2 between a nerve fiber layer and a Ganglion cell layer(GCL), a photoreceptor inner segment outer segment (ISOS) L3, a retinalpigment epithelium (RPE) L4, a Bruch membrane (BM) L5, and a choroid L6.The tomographic image has an axis of abscissa (main-scanning directionof the OCT) as an x-axis and an axis of ordinates (vertical directionthereof) as a z axis.

FIG. 2C illustrates a fundus image captured by the fundus imagingapparatus 400. The fundus imaging apparatus 400 is configured to capturea fundus image of the eye and may be a fundus camera or an SLO (ScanningLaser Ophthalmoscope), for example. FIG. 2C illustrates a macular area Mand an optic disk D and thick lines representing blood vessels. Thefundus image has an axis of abscissa (main-scanning direction of theOCT) as an x-axis and an axis of ordinates (sub-scanning direction ofthe OCT) as a y axis. The tomographic imaging apparatus 200 and thefundus imaging apparatus 400 may be provided integrally or separately.

The image processing apparatus 300 includes an image capturing unit 301,a storage unit 302, an image processing unit 303, an instructing unit304, and a display control unit 305. The image capturing unit 301includes a tomographic image generation unit 311. The image processingapparatus 300 may include one or more processors and one or morememories, not illustrated, for example. The one or more processors mayexecute programs stored in the one or more memories so that the one ormore processors can function as the image capturing unit 301, the imageprocessing unit 303, the instructing unit 304, and the display controlunit 305. Each of the processors may be a hardware module such as a CPUand a GPU. For example, the image processing apparatus 300 obtainssignal data of a tomographic image captured by the tomographic imagingapparatus 200 and performs signal processing thereon to generate atomographic image. For example, the image capturing unit 301 acquires aplurality of two-dimensional tomographic images based on measurementlight controlled for scanning an identical position of the eye target toaveraging of tomographic images thereof. Fundus image data captured bythe fundus imaging apparatus 400 are also acquired. The generatedtomographic images and the fundus images are stored in the storage unit302. The image processing unit 303 includes a detection unit 331, afirst alignment unit 332, a second alignment unit 333, an imageselection unit 334, a third alignment unit 335, and an image compositionunit 336.

The detection unit 331 is configured to detect a boundary line betweenlayers from the retina. The first alignment unit 332 is configured toperform alignment in the horizontal direction (x-axis) of the retina.The second alignment unit 333 is configured to perform alignment in thevertical direction (z-axis) of the retina. The image selection unit 334is configured to select a reference tomographic image for an alignmentfrom a plurality of tomographic images and to select tomographic imagesto be averaged. The third alignment unit 335 is configured to set aplurality of areas for an alignment in a characteristic part within atomographic image and at the same time to perform alignment area by areain the horizontal direction (x-axis) and the vertical direction(z-axis). The image composition unit 336 is configured to average thetomographic images selected by the image selection unit 334.

The external storage unit 500 is configured to hold, in association,information regarding a target's eye (such as the name, age and sex ofthe patient), captured image data, imaging parameters, image analysisparameters, and parameters set by an operator.

The input unit 700 may be a mouse, a keyboard, or a touch operationdisplay screen, and an operator can instruct the image processingapparatus 300, the tomographic imaging apparatus 200, and the fundusimaging apparatus 400 through the input unit 700.

Next, with reference to FIGS. 3A and 3B, process procedures to beperformed by the image processing apparatus 300 according to thisembodiment will be described. FIG. 3A is a flowchart illustrating a flowof an operation process in the entire system according to thisembodiment, and FIG. 3B is a flowchart illustrating a flow of ahigh-quality image generation process according to this embodiment.

Step S301

In step S301, a target's eye information obtaining unit, notillustrated, externally obtains a target identification number asinformation for identifying a target's eye. Based on the targetidentification number, information regarding the target's eye held in anexternal storage unit 500 is obtained and is stored in the storage unit302.

Step S302

In step S302, the target's eye is scanned for imaging. In order to scana target's eye, an operator selects a scan start, not illustrated, andthe tomographic imaging apparatus 200 controls the drive control unit202 to operate the galvanometer mirror 201 and scan a tomographic image.The galvanometer mirror 201 includes an X scanner in a horizontaldirection and a Y scanner in a vertical direction. When the directionsof these scanners are changed, scanning can be performed in thehorizontal direction (X) and the vertical direction (Y) in an apparatuscoordinate system. By changing the directions of the scannerssimultaneously, scanning can be performed in a direction combining thehorizontal direction and the vertical direction. Thus, scanning can beperformed in an arbitrary direction on a fundus plane.

For imaging, imaging parameters are to be adjusted. More specifically,the position of the internal fixation lamp, a scan range, a scanpattern, a coherence gate position, and a focus are at least set. Thedrive control unit 202 controls light emitting diodes in the displayunit 241 to control the position of the internal fixation lamp 204 so asto image a macular area center or an optic disk. The scan patterndefines a scan pattern such as a raster scan, a radial scan, and a crossscans for imaging a three-dimensional volume. With any of scan patternsselected, a plurality of images of one line is captured repetitively(where the number of times of repetition is two or more). According tothis embodiment, a case will be described where the scan pattern is across scan and one identical position is to be imaged 150 timesrepetitively. After completion of the adjustment of the imagingparameters, an operator may select to start imaging, not illustrated,for imaging. According to the present disclosure, the tomographicimaging apparatus 200 tracks the target's eye for imaging an identicallocation (one line) for averaging to scan the target's eye with areduced influence of involuntary eye movements during fixation, thoughthe detail descriptions will be omitted.

Step S303

In step S303, a tomographic image is generated. The tomographic imagegeneration unit 311 performs general reconstruction processing oninterference signals to generate a tomographic image.

First, the tomographic image generation unit 311 performs fixed patternnoise removal from the interference signals. The fixed pattern noiseremoval averages a plurality of detected A-scan signals, extracts fixedpattern noise, and subtracts it from input interference signals. Next,the tomographic image generation unit 311 performs a desired windowfunction process to optimize a depth resolution and a dynamic rangehaving a trade-off relationship when Fourier transform is performed in afinite interval. Next, an FFT process is performed to generate a faultsignal.

Step S304

In step S304, the image processing unit 303 performs composed imagegeneration. Processing to be performed by the image processing unit 303will be described with reference to the flowcharts in FIG. 3B, FIG. 4,and FIG. 5 and FIGS. 6A to 6C to FIGS. 13A and 13B.

Step S341

In step S341, the detection unit 331 detects a boundary line of theretina layer in the plurality of tomographic images captured by thetomographic imaging apparatus 200. The detection unit 331 detects one ofboundaries L1 to L6 in the tomographic image in FIG. 2B. A median filterand a Sobel filter are applied to tomographic images to be processed togenerate images (hereinafter, called a median image and a Sobel image).Next, a profile is generated for each A-scan from the generated medianimage and Sobel image. The median image corresponds to a luminance valueprofile, and the Sobel image corresponds to a gradient profile. Then,peaks are detected within the profile generated from the Sobel image.With reference to profiles of the median images before and after thedetected peaks and between peaks, boundaries of the areas of the retinalayer are detected.

Step S342

In step S342, between the plurality of tomographic images, an alignmentin the horizontal direction (x-axis) of the retina and an alignment inthe depth direction (z-axis) of the retina are performed thereon. Itshould be noted that the alignments here will be described as globalalignment. A method for the global alignment in S342 will be describedwith reference to the flowchart in FIG. 4.

Step S3421

In step S3421, a two-dimensional matrix is initialized which is forstoring alignment parameters for performing alignments on thetomographic images. Each matrix has elements each storing informationfor increased image quality such as deformation parameters and imagesimilarities for alignments.

Step S3422

In step S3422, the first alignment unit 332 generates a line projectedimage. The line projected image is an example of a projected image. Theline projected image and a generation method thereof will be describedwith reference to FIGS. 6A and 6B. FIG. 6A illustrates a schematicdiagram of one tomographic image. The tomographic image has boundarylines L1 to L4, an A-scan AS, and VESSEL indicating a blood vessel.Referring to FIG. 6A, the blood vessel is indicated by an area enclosedby two vertical lines, and a deep layer part is shaded by the bloodvessel in a shallow layer part. The term “line projected image” refersto an image acquired by averaging tomographic images in the z-axisdirection in A-scans. FIG. 6B illustrates a schematic diagram of a lineprojected image generated from one tomographic image. According to thisembodiment, a plurality of tomographic images of an identical positionis captured for high-quality processing by using the plurality oftomographic images. FIG. 6C illustrates an example of a line projectedimage generated from tomographic images acquired by imaging an identicala plurality of number of times. FIG. 6C has an axis of abscissacorresponding to the X-axis of a tomographic image and an axis ofordinates corresponding to a time axis indicated by t. A tomographicimage has a lower luminance value in a lower part of the blood vesselbecause of its shadow, compared with surrounding parts. Accordingly, theline projected image has a lower luminance value corresponding to theblood vessel.

Step S3423

In step S3423, alignment targets are selected. According to thisembodiment, all tomographic images are set as reference images, andalignments are performed between the set reference tomographic imagesand the remaining tomographic images.

Accordingly, in step S3423, a tomographic image with Index 0 is set as areference, and alignments are performed between Index 0 and Indices 1 to149. Next, alignments are performed between a reference tomographicimage with Index 1 and Indices 2 to 149. Next, alignments are performedbetween a reference tomographic image with Index 2 and Indices 3 to 149.These processes are repeated. The repetition is judged in step S3428,which will be described below.

When the index of the reference image is moved up by one, the startindex of the images being an alignment target is also moved up by one.This will be described with reference to a case where a tomographicimage with Index 2 is set as a reference image. When Index 2 is set as areference, the alignment between Index 0 and Index 1, Index 0 and Index2, and Index 1 and Index 2 have already undergone an alignment by theprocesses up to that point. Therefore, when a tomographic image withIndex 2 is set as a reference, an alignment may start from Index 3.Thus, for alignments between all tomographic images, half combinationsthereof may be calculated.

Step S3424

In step S3424, the first alignment unit 332 performs an alignment in ahorizontal direction (x-axis) on the retina between the plurality oftomographic images. Here, the alignment in the horizontal directionapplies line projected images generated in step S3422. The alignmentapplying line projected images corresponds to an example of an alignmentaccording to a first method. The line projected image alignments will bedescribed with reference to FIGS. 7A and 7B. FIG. 7A illustrates anexample of a line projected image generated from tomographic image groupacquired by a plurality of imaging operations performed on an identicalposition. Referring to FIG. 7A, focusing on a blood vessel part, aslight difference may be found as a result of tracking of the target'seye during an imaging operation. Accordingly, the positional differenceis to be corrected. Although image data is used here, image data targetsfor alignments are one-dimensional data, as illustration in FIG. 7B.FIG. 7B illustrates line projected images Index 0 and Index 1 having animage size of 1024×1. With reference to Index 0, an image of a range ofROI of Index 1 undergoes an alignment. The ROI has a size smaller thanthe image size in the horizontal direction and has an equal size in thevertical direction. For example, the ROI may have a size of 1000×1. Thealignment processing to be performed between images may include, forexample, pre-defining an evaluation function representing a similaritybetween two line projected images, calculating the evaluation value forchanged line projected image positions, and determining a locationhaving the highest evaluation value as an alignment result. In otherwords, a moving amount to a location having the highest evaluation valueis defined as a positional deviation amount between the line projectedimages for performing an alignment. The evaluation function may be amethod for evaluating a pixel value (such as a method for evaluating byusing a correlation coefficient).

Expression 1 is an expression in a case where a correlation coefficientis used as an evaluation function representing a similarity.

$\begin{matrix}\frac{\underset{S}{\int\int}\left( {{f\left( {x,z} \right)} - \overset{\_}{f}} \right)\left( {{g\left( {x,z} \right)} - \overset{\_}{g}} \right){dxdz}}{\sqrt{\underset{S}{\int\int}\left( {{f\left( {x,z} \right)} - \overset{\_}{f}} \right)^{2}{dxdz}\underset{S}{\int\int}\left( {{g\left( {x,z} \right)} - \overset{\_}{g}} \right)^{2}{dxdz}}} & \left\lbrack {{Expression}\mspace{14mu} 1} \right\rbrack\end{matrix}$

In Expression 1, f(x,z) is an area of a first line projected image, andg(x,z) is an area of a second line projected image.

f,g   [Expression 2]

represents an average of the areas f(x,z) and the area g(x,z). The term“area” here refers to an image area to be used for an alignment, and thearea has a size equal to or smaller than that of a normal line projectedimage and is set as an ROI size as described above.

The evaluation function is not limited to the one described above butmay be a SSD (Sum of Squared Difference) or a SAD (Sum of AbsoluteDifference) for evaluating a similarity or a difference between images.

Alternatively, the alignment may be performed based on a method such asPOC (Phase Only Correlation).

Because an alignment is performed between line projected images, theimage alignment is performed one-dimensionally and can be performed in ahorizontal direction in a quick and stable manner. Informationindicating whether there are similar image features or not can becalculated.

Step S3425

In step S3425, the boundary line information detected in step S341 isobtained. The boundary line to be obtained here may be boundary lineinformation on tomographic images that are current alignment targets.For example, boundary line information between Index 0 and Index 1 maybe obtained.

Step S3426

In step S3426, the second alignment unit 333 performs an alignment in adepth direction (z-axis) on the retina. Here, the boundary lineinformation is used for an alignment in the depth direction. Thealignment applying boundary information corresponds to an example of analignment according to a second method to be executed at a differentclock time from that of the alignment according to the first method. Theboundary line information alignment will be described with reference toFIGS. 8A to 8C. FIG. 8A illustrates an example of a boundary line to beused for an alignment. According to this embodiment, a boundary line L1(ILM) and a boundary line L3 (ISOS) are used.

FIG. 8A illustrates a reference image Index 0 and a target imageIndex 1. The reference image and the target image undergo an alignmentin a horizontal direction in step S3424. Accordingly, an alignment in adepth direction by using the reference boundary lines L1, L3 andalignment target boundary lines L1′, L3′ corrected with a horizontalalignment parameter are used for an alignment in the depth direction. Itshould be noted that the layer boundary to be used for an alignment isnot limited to the layer boundary as described above, and the number oflayer boundaries to be used is not limited to two. The alignment usingthe boundary information corresponds to an example of the alignmentaccording to the second method to be executed at a different clock timefrom that of the alignment according to the first method.

Referring to FIG. 8B, the reference image boundary lines L1, L3 and thealignment target boundary lines L1′, L3′ are displayed simultaneouslyfor simple description. Referring to FIG. 8B, the boundary lines aredivided into N in the vertical direction. This results in Area 0 to AreaN−1. Referring to FIG. 8B, the entire image is actually divided intoareas though the image at the center is not divided for clearillustration. A vertical arrow Difference 1 represents a difference(positional deviation amount) between L1 and L1′, and a vertical arrowDifference 3 represents a difference between L3 and L3′. Thesedifferences are acquired for each of the Area 0 to Area N−1. Although,for simplicity, the boundary lines in the horizontal direction have anequal size according to this embodiment, the retina layer in reality maydeviate upward (in the direction O of the z-axis) in an image, and apartial area of the retina layer may be missing from the image. In thiscase, a boundary line cannot be detected in the entire image.Accordingly, a range in which boundary lines between the reference imageboundary lines L1, L3 and the alignment target boundary lines L1′, L3′may be divided into N for alignment between the boundary lines. Thefollowing description assumes that the number N of divisions is equal to12. The number N of divisions may be changed in accordance with theimage size in the horizontal direction. Alternatively, the number N ofdivisions may be changed in accordance with the size of the horizontalwidth of a commonly detected boundary line.

FIG. 8B illustrates averages D_(0t) to D_(11t) of Difference 1 andDifference 3 in each of the areas. In other words, the average ofdifferences between ILM and ISOS is handled as a representative value ofthe differences in the area. Next, the representative values D_(0t) toD_(11t) acquired in each of the areas are sorted in increasing order. Mrepresentative values in increasing order of the sorted representativevalue are used to calculate an average and a variance. The followingdescription according to this embodiment assumes that M is equal to 8.However, the number is not limited thereto. The number of Ms may belower than the number of Ns (M<N). The average and the variance arecalculated by shifting the sorted representative values by one. In otherwords, according to this embodiment, because eight representative valuesof 12 divided areas are used for the calculation, five average valuesand five variance values can be acquired in total. Next, an averagevalue corresponding to the minimum variance value among the fivevariance values is defined as a shift value in the depth direction. Forexample, in a case where the minimum variance value is Var0 among fivevariance values, the shift value in the depth direction is Avg0. In thismanner, an area for boundary line alignment may be divided, and a valueproducing a minimum variation among combinations of difference values ofthe divide areas may be used, which prevents use of areas causingimproper boundary line detection. Therefore, the shift value in thedepth direction can be calculated in a stable manner. Having describedthat an average value is used as the shift value in the depth direction,a median value may be used instead. In other words, a representativevalue may be used. Furthermore, having described a variance value isused as a value indicative of a variation, a standard deviation or anindex which can evaluate a variation of a value can be used. Havingdescribed the example that the representative values are sorted inincreasing order, the representative values may be sorted in decreasingorder. The second alignment unit 333 in this way is configured to sortpositional deviation amounts of a plurality of areas in order based onthe values of the positional deviation amounts, sequentially select apredetermined number of combinations of positional deviation amounts inincreasing or decreasing order, calculate values indicative ofvariations of the combinations, and select a combination of positionaldeviation amounts having a minimum variation value.

Finally, the acquired shift value in the depth direction is used toshift the entire boundary lines in the Z direction. FIG. 8C illustratesthe example. Again, difference values in the Area 0 to Area N−1 arecalculated. Representative values D₀ to D₁₁ of the difference valuesacquired in the areas are sorted in increasing order. M of sortedrepresentative values are used from the lowest value to calculate anaverage value. The average value will be called global Distance. Thus, afinal positional deviation amount (global Distance) between the boundarylines after alignment of the reference image and the target image in theX direction and the Z direction is determined.

The number N of divisions and the number M of selections for calculatinga shift value in the depth direction are assumed to be equal to numbersN and M of divisions for calculating the final positional deviationamount (global Distance) in the description above. However, embodimentsof the present disclosure are not limited thereto. These numbers may bedifferent or identical. For example, the number N of divisions forcalculating the shift value may be equal to 12, the number M ofselections may be equal to 8, the number N of divisions for calculatinga positional deviation amount (global Distance) may be equal to 15, andthe number M of selections may be equal to 10.

Referring to FIG. 8C, the boundary lines are shifted for simpleillustration. However, data may not actually be shifted, but thedifference value of each area may be calculated in consideration of theshift value in the depth direction.

Step S3427

In step S3427, values for alignment and image similarity initialized instep S3421 are stored in a two-dimensional matrix for storing parametersfor higher image quality. For example, in a case where a reference imagehas Index 0 and a target image has Index 1, an alignment parameter X inthe horizontal direction, an alignment parameter Z in the depthdirection, representative values D₀ to D₁₁ of difference values ofareas, global Distances and image similarities are stored in an element(0, 1) of a two-dimensional matrix. In addition to these kinds ofinformation a false flag as a result of a boundary line detection may bestored in association with a darker image having a boundary line that isdetected clearly due to a blink. On the other hand, if the layerboundary detection succeeds, a true flag may be stored in associationwith the image. In a case where the boundary line alignment includesalignment by rotating a boundary line itself, the turn angle and thecenter coordinates of the rotation axis may further be stored, detailsof which will not be described in this description of this embodiment. Amagnification may further be store if a magnification correction is tobe corrected.

Step S3428

In step S3428, whether alignment has been performed on all images withthe remaining target image being a reference image or not is determined.If all images are not processed as the reference images, the processingreturns to step S3423. If so, the processing moves to step S3429.

Step S3429

In step S3429, the remaining elements of the two-dimensional matrix areupdated. The process above calculates half of the combinations as instep S3423. For that reason, those values are copied to the elementsthat have not been calculated. For example, parameters in an element(0, 1) in the two-dimensional matrix are copied to an element at (1, 0).In other words, an element (i, j) is copied to an element (j, i). Inthis case, because the alignment parameters X and Z and the turn angleare reversed, they are copied after being multiplied by a negativevalue. Because an image similarity, for example, is not reversed, thesame value is copied as it is.

These processes are performed in the global alignment. In the flowabove, a Z-direction alignment is performed after an X-directionalignment, but an X-direction alignment may be performed after aZ-direction alignment. Alternatively, a Z-direction alignment may beperformed after an X-direction alignment, and, again, a Z-directionalignment may be performed after an X-direction alignment. In thismanner, alignments may be repeated several times. The process flow nowreturns to FIG. 3B.

Step S343

In step S343, the image selection unit 334 selects a reference image.The reference image selection is performed based on a result of theglobal alignment performed in step S342. In step S342, a two-dimensionalmatrix is generated, and information for generating a high quality imageis stored in elements of the matrix. Thus, the information is used toperform the reference image selection. In the reference image selection,in a case where a boundary line detection flag is stored, theinformation is used, and the image similarity (that is an example of aresult of a comparison between projected images) acquired in thehorizontal direction alignment and representative values D₀ to D₁₁ ofthe difference values of the areas acquired in the depth directionalignment. More specifically, images with Index 0 are counted within arange of elements (0, 0) to (0, 149) in the two-dimensional matrix,having True as a boundary line detection flag, having an imagesimilarity equal to or higher than a threshold value, and havingrepresentative values D₀ to D₁₁ of the difference value acquired for theareas within a threshold value (or equal to or lower than thepredetermined threshold value) in eight or more areas of 12 areas. Also,images with Index 1 are counted within a range of elements (1, 0) to (1,149) in the two-dimensional matrix, having True as a boundary linedetection flag, having an image similarity equal to or higher than athreshold value, and having representative values D₀ to D₁₁ of thedifference value acquired for the areas within a threshold value ineight or more areas of 12 areas. These processes are performed on all ofthe images. It should be noted that one of the image similaritiesacquired in the horizontal direction alignment and representative valuesD₀ to D₁₁ of the difference values acquired for the areas in the depthdirection alignment may be used.

Next, examples will be described with reference to FIGS. 9A and 9B inwhich the conditions of the image similarities and the representativevalues D₀ to D₁₁ of the difference value acquired for the areas aresatisfied. FIG. 9A illustrates an example in which the conditions of theimage similarities and the representative values D₀ to D₁₁ of thedifference values acquired for areas being equal to or lower than athreshold value in eight or more areas of 12 areas are simultaneouslysatisfied, which is shaded in FIG. 9A. FIG. 9B illustrates a total ofthe number satisfying the condition of the image similarities and thenumber satisfying the condition that the representative values D₀ to D₁₁of the difference values acquired for areas being equal to or lower thana threshold value in eight or more areas of 12 areas, which is shaded inFIG. 9B. According to this embodiment, both of the conditions aresimultaneously satisfied as illustrated in FIG. 9A. However, embodimentsof the present disclosure are not limited thereto. As illustrated inFIG. 9B, each of conditions may be independently satisfied, and thetotal of the numbers may be counted. The reference image selection isnot limited to the numerical values of eight or more areas of 12 areasbut may be based on other numerical values.

Then, tomographic images the number of which is at a maximum under theconditions are selected as reference images. In other words, in a casewhere the image selection unit 334 compares, among of a plurality oftwo-dimensional tomographic images, one two-dimensional tomographicimage and a plurality of other two-dimensional tomographic images, theimage selection unit 334 selects reference images based on the number oftwo-dimensional tomographic images having similarities equal to orhigher than a predetermined threshold value and having positionaldeviation amounts of layer boundaries equal to or lower than apredetermined threshold value. More specifically, the image selectionunit 334 selects reference images based on, in a case where onetwo-dimensional tomographic image and a plurality of othertwo-dimensional tomographic images are compared among a plurality oftwo-dimensional tomographic images, the number of two-dimensionaltomographic images having similarities equal to or higher than apredetermined threshold value and the number of areas equal to or higherthan a predetermined number where the areas are a plurality of areasdividing the two-dimensional tomographic images in the horizontaldirection having positional deviation amounts of layer boundaries equalto or lower than a predetermined threshold value.

It should be noted that in a case where there are images having an equalnumber satisfying the conditions, a reference image may be selectedbased on another evaluation value such as one with an image similarityhaving a highest total or average value, one with a highest total numberof difference values with their representative values D₀ to D₁₁ acquiredfor each of the areas equal to or lower than a threshold value, one witha lowest total value or average value of global Distance, one with adetected boundary line having a maximum length or one with a boundaryline in the depth direction positioned closer to the center. One or aplurality of these conditions may be applied. They may be used not onlyas a criteria for narrowing in a case where there are the images havingan equal count satisfying the conditions but also for counting imagessatisfying the conditions for the reference image selection.

Step S344

In step S344, the image selection unit 334 may select an addition image.The addition image selection includes judging whether another imagesatisfies a condition with respect to the reference image acquired instep S343. For addition image selection, an image selection flag may beset, and if a condition is satisfied, True is set thereon, and if not,False is set thereon. As conditions for the addition image selection,the number of images having True as the boundary line detection flag, animage similarity equal to or higher than a threshold value, and havingrepresentative values D₀ to D₁₁ of difference values of the areas withina threshold value equal to or more than eight areas of 12 areas, forexample, like step S343. It should be noted that different thresholdvalues may be applied between the addition image selection and thereference image selection. For example, a tight threshold valuecondition may be applied for the reference image selection while alooser condition may be applied for the addition image selection thanthat for the reference image selection. Alternatively, an equalcondition may be used for the reference image selection and the additionimage selection.

FIGS. 10A to 10C illustrate examples of line projected images generatedby using tomographic images selected after alignment by the firstalignment unit 332, the second alignment unit 333, and the imageselection unit 334. FIG. 10A illustrates a line projected imagegenerated from a tomographic image before an alignment, FIG. 10Billustrates a line projected image generated from a tomographic imageafter an alignment, and FIG. 10C illustrates a line projected imagegenerated by using a tomographic image selected after an alignment.Focusing on a feature of the blood vessel, FIG. 10B has more linearlyuniform blood vessels than that of the FIG. 10A. Referring to FIG. 10C,a part having a locally deviated blood vessel is excluded, and an imageacquired by capturing a substantially identical position is onlyselected.

Step S345

In step S345, the third alignment unit 335 sets a plurality of areas foralignment in a part having a tomographic image internal feature betweena reference image and a plurality of selected tomographic images, andalignments in the horizontal direction (x-axis) and in the depthdirection (z-axis) are simultaneously performed on the retina in areas.The alignments here will be described as local alignments. A method tobe applied in the local alignment in S345 to be performed by the thirdalignment unit 335 will be described with reference to the flowchart inFIG. 5.

Step S3451

In step S3451, a process to be performed is determined based on whethera given image has already been selected in step S344 or not. If theimage selection flag has True, the processing moves to step S3452. Ifthe image selection flag has False, the processing moves to step S3459.

Step S3452

In step S3452, a mask image is generated for an averaging process. Themask image, for example, may have all pixels having a value of 1.0.According to this embodiment, all values are equal. However, embodimentsof the present disclosure are not limited to use of a mask image.Weighting may also be performed in accordance with a location. Forexample, vertical and horizontal peripheral areas (about several pixels)of an image may have a value lower than 1.0. In this case, the valuesmay not be lowered uniformly, but the value may be lowered gradually asthey go from the center to ends of the image.

Step S3453

In step S3453, image deformation is performed in tomographic images(B-scans). This will be described with reference to FIGS. 11A and 11B.FIG. 11A illustrates examples of a reference image and a reference maskimage, and FIG. 11B illustrates a target image 1 and target mask image 1acquired by shifting the images in FIG. 11A to the lower rightdirection. As illustrated in FIGS. 11A and 11B, tomographic imagesexcluding the reference image are deformed based on the alignmentparameters X and Z acquired by the global alignment in step S342. Theblack areas illustrated in FIG. 11B are invalid areas caused by theshift of the images in the X and Z directions as a result of thealignments. Here, because no image data are present, the value is equalto 0. Not only the tomographic image but also the mask image is deformedwith the same parameter.

Step S3454

In step S3454, the boundary line information detected in step S341 isobtained. It should be noted that the boundary line to be obtained hereis boundary line information only of the current alignment referencetomographic image. According to this embodiment, the boundary lines L1and L3 are acquired.

Step S3455

In step S3455, an alignment area is set such that it includes a featurearea of the target image. This will be described with reference to FIG.12.

FIG. 12 illustrates a reference image and a target image 1. The targetimage 1 has a plurality of alignment areas (or ROIs (regions ofinterest)) set based on the boundary line information L1 and L3 of thereference tomographic image. The size in the depth direction of the ROIis set wider in the upper and lower directions by about several 10pixels with reference to the lines L1 and L3. In a case where parametersof about several 10 pixels in the upward and lower directions are set,the result of the global alignment may be used to correct the parameter.In a case where the entire image is shifted toward the lower directionin the global alignment as illustrated in the target image 1 in FIG. 12,there is an invalid area in an upper end part of the image. In thiscase, the initial size of the ROI may be corrected so as not to includethe range for setting the ROI and a search area therefor. The size inthe horizontal direction of the ROI is set from a size dividing theimage into K. The number K of divisions is set in accordance with animaging parameter such as the size of the image (the number of A-scans)or the size of image capturing (10 mm). For example, according to thisembodiment, in a case where the number of A-scans is equal to 1024 andthe size of image capturing is equal to 10 mm, the number K of divisionsis equal to 10. The set values for the size in the horizontal directionand for an ROI are also corrected by using the result of the globalalignment. Because there may be an invalid area in the horizontaldirection, like the parameters in the vertical direction, a range forsetting the ROI and a search area therefor may be set so as not toinclude the invalid area.

The ROIs for a local alignment are set to superimpose on each other.This is because when ROIs are not overlapped and the size of theresulting ROI is small, there may be a location in the ROI which doesnot include a characteristic area. For example, when the retina isimaged with a narrow angle of view (about 3 mm), the captured image mayhave flat tissue in a wide range. On the other hand, in a case whereROIs are not overlapped and the range of the resulting ROI is set widerto include a feature, a smaller number of samples are obtained for alocal alignment, which results in a rough alignment. In order to solvethese problems, the size in the X direction of the ROI is increased, andROIs are superimposed on each other for setting. Although FIG. 12 doesnot render an ROI at the center of the image, an ROI is actually set onthe retina from the left end to the right end of the image. Theintervals for setting ROIs may be determined in consideration of asearch range for an ROI alignment. More specifically, in a case where ahorizontal search range for ROI alignment is XR, the interval betweencenter coordinates of neighboring ROIs is set to be equal to or longerthan 2XR. This is because there is a possibility that the centerpositions of the neighboring ROIs may change place with other if theinterval between the center coordinates is shorter than 2XR. In thisway, the positional relationship in the horizontal direction between aplurality of ROIs (first area and second area) is set based on a searchrange for an ROI alignment.

Step S3456

In step S3456, an ROI is used for an area alignment. The area alignmentis performed on images. Accordingly, Expression 1, like the lineprojected image alignment in step S3424, is used for performing analignment based on an image similarity. However, the evaluation valueregarding a similarity is not limited thereto, but an SSD (Sum ofSquared Difference), an SAD (Sum of Absolute Difference) or the like maybe used. Alternatively, an alignment may be performed based on a methodsuch as POC (Phase Only Correlation).

An image alignment searches whether an ROI set in a target image islocated in a reference image. In this case, because the result of theglobal alignment performed in step S342 is used to deform thetomographic image in step S3453, the reference image and the targetimage are substantially in alignment. Thus, in the search range foralignment of a reference image, a search may be performed from theinitial position of an ROI by the vertical and horizontal severalpixels. Then, the location having the most similar pixels is determinedas an alignment result. Weighting in accordance with a location may beperformed in calculation of evaluation values for similarities for localalignment. In this case, the center of the search range may be mosthighly weighted (such as a weight of 1.0), and the weight may be reduced(such as a weight of 0.9) as the distance to the outside of the searchrange increases. Such a weight may be changed pixel by pixel smoothly.Thus, when there is a slight difference between evaluation values, theinitial position is selected.

The search range in an ROI may be fixed or may be varied in accordancewith the imaging angle of view, the region to be imaged, or an imagelocation (end or center).

Step S3457

In step S3457, alignment parameters acquired in step S3456 areinterpolated. This will be described with reference to FIGS. 13A and13B. FIG. 13A illustrates ROI 1 to ROI 3 in an initially set area.Inverted triangles C1 to C3 indicate center positions of the ROI 1 toROI 3. FIG. 13B illustrates an example of moved ROIs after the alignmentin step S3456. FIG. 13B illustrates a case where the ROI 1 and the ROI 3are moved to the right and the ROI 2 is not moved. Thus, the centers C1and C3 of the ROIs are moved to C1′ and C3′, respectively. A movingamount of an A-scan may be calculated from moving amounts of ROIs basedon moving amounts of the center positions of a neighboring ROI and anROI. For example, the center position of the ROI 1 is moved from C1 toC1′, and the center position of the ROI 2 stays at C2. Here, the Xdirection moving amount of each A-scan between C1 before the deformationto C2 can be acquired by the following Expression 3.

$\begin{matrix}{{W = {1.0 - \frac{\left( {A_{\_ \; {before}} - {X\; 1}} \right)}{\left( {{X\; 2} - {X\; 1}} \right)}}}{{TransX} = {{\Delta \; X\; 1*W} + {\Delta \; X\; 2*\left( {1.0 - W} \right)}}}{A_{\_ \; {after}} = {A_{\; {\_ \; {before}}} - {TransX}}}} & \left\lbrack {{Expression}\mspace{14mu} 3} \right\rbrack\end{matrix}$

In Expression 3, X1 and X2 are initial center coordinates of ROIs, ΔX1and ΔX2 are X direction moving amounts of the center coordinates of theROIs, A_before is a value of an A-scan index before a deformation, andA_after is a value of the A-scan index before the deformation which isreferred by A_before. In a case where, for example, A_before is 55 andA_after is 56 as a result of the calculation, the A-scan index 55contains A-scan data of the A-scan index 56. Expression 3 expresses thata weight for the moving amount varies in accordance with the distancesbetween a plurality of ROIs in an area where the plurality of ROIsoverlap. More specifically, in an area where a plurality of ROIsoverlap, a weight for a moving amount of the center coordinate of theclosest ROI of a plurality of ROIs is higher than a weight for themoving amount of the center coordinate of a farther ROI of the pluralityof ROIs.

The moving amount in the Z direction can also be acquired from themoving amounts of the center positions of the ROIs based on the samemanner as that of Expression 3, and data move by several pixels in thevertical direction. A_after may have a value that is a real number or aninteger. If it is a real number, new A-scan data are generated by aknown interpolation method (such as Bilinear or Bicubic) using aplurality of A-scan data pieces. If it is an integer, data of thecorresponding A-scan index is referred as it is.

The third alignment unit 335, as described above, corresponds to anexample of a determination unit configured to determine a moving amountof a second two-dimensional tomographic image to a first two-dimensionaltomographic image in an area having a first area and a second areaoverlapped based on a positional deviation amount of the first area anda positional deviation amount of the second area.

Step S3458

In step S3458, each A-scan is moved in the X direction and in the Zdirection based on the A-scan moving amount acquired in step S3457.Thus, a tomographic image acquired by deforming each A-scan can begenerated. It should be noted that not only a tomographic image but alsoa mask image are deformed with an equal parameter.

Step S3459

In step S3459, whether all tomographic images have undergone a localalignment with respect to a reference image or not is determined. If allimages have not been processed, the processing returns to step S3451. Ifall images have undergone the local alignment, the local alignmentprocessing completes.

This processing is performed by the local alignment. Next, theprocessing flow returns to FIG. 3B.

Step S346

In step S346, the image composition unit 336 averages a referencetomographic image selected by the image selection unit 334 and aplurality of tomographic images. This averaging process holds, for eachpixel, a sum value SUM_A of values acquired by multiplying a pluralityof tomographic images by a mask image value and a sum value SUM_B of aplurality of mask image values. Because a mask image stores 0 as aninvalid area without having image data because of an alignment, the sumvalue SUM_B of mask image hold values different from each other betweenpixels. It is generally assumed that images are moved by several 10pixels vertically and horizontally because of an alignment. Thus, in acase where N images are used to overlap, the pixel value of SUM_B nearthe image center is N, and the pixel value of SUM_B of an image end parthas a value lower than N.

In this averaging process, SUM_A may be divided by SUM_B to acquire atomographic image acquired by averaging.

Step S305

In step S305, the detection unit 331 performs a boundary line detectionon a high quality tomographic image generated in step S304. The boundaryline detection can be performed according to the same method as that instep S341, for example.

Step S306

In step S306, the result of the high quality tomographic image generatedby averaging is displayed on the display unit 600.

Step S307

In step S307, an instruction obtaining unit, not illustrated, externallyobtains an instruction to complete the imaging of tomographic images byusing the image processing system 100 or not. The instruction is inputby an operator through the input unit 700. If an instruction to completethe processing is obtained, the image processing system 100 ends theprocessing. On the other hand, in order to continue the imaging withoutending the processing, the process returns to step S302 where theimaging is continued. In this manner, the process is performed by theimage processing system 100.

With the aforementioned configuration, according to this embodiment, alocal alignment is performed by using boundary lines and image featurevalues, and, at the same time, a reference image and an addition imageare selected for high-quality image generation. Furthermore, for aplurality of two-dimensional tomographic images having undergone analignment, similarities between corresponding local areas arecalculated, and an alignment process is performed area by area. Thus,images for generating a high quality image can be selected. Then, in acase where the retina layer is locally deformed due to, for example, aninvoluntary eye movements during fixation, alignments are performeddivision by division of the image. Therefore, a high qualitytwo-dimensional tomographic image can be generated.

Variation Example 1

According to this embodiment, the first alignment unit 332 generates aline projected image and performs an alignment in a horizontal direction(x-axis) on the retina based on an image similarity between lineprojected images. However, the horizontal direction (x-axis) alignmentis not limited thereto. For example, the first alignment unit 332 maydetect an edge from a line projected image and may perform an alignmentby using a line edge image having the detected edge. Alternatively, thefirst alignment unit 332 may detect a feature point from a lineprojected image by using a method such as SIFT (Scale-Invariant FeatureTransform) or AKAZE (Accelerated KAZE) and may perform an alignmentbased on the detected feature point.

Variation Example 2

According to this embodiment, the second alignment unit 333 usesboundary line information to perform an alignment in the depth direction(z-axis) on the retina, for example. However, the alignment is notlimited to the depth direction (z-axis) alignment. For example, thesecond alignment unit 333 may generate a horizontal line projected imageacquired by averaging a plurality of A-scans in the X-axis direction andmay perform an alignment in the depth direction based on the generatedprojected image. It should be noted that the alignment in this case mayuse the method according to Embodiment 1 or the method according toVariation Example 1. A horizontal projected image may be generated byusing tomographic images as they are imaged. However, a slope may occurin the retina layer in accordance with the state of the imaging.Therefore, the image may be corrected. The correction method in thatcase may include image deformation such that the shape of the boundaryline L1 (ILM) or the boundary line L3 (ISOS) can be flat, for example.FIGS. 14A to 14D illustrate examples of the image correction in thatcase. FIG. 14A illustrates a tomographic image before a deformation onits left side and illustrates a horizontal line projected imagegenerated from the tomographic image before a deformation on its rightside. FIG. 14B illustrates a tomographic image and a horizontal lineprojected image after a deformation for making L1 to be flat, and FIG.14C illustrates a tomographic image and a horizontal line projectedimage after a deformation for making L3 to be flat. For an alignment inthe depth direction, a reference location for a boundary line to bedeformed during the image deformation operation is not changed, butother locations are deformed. FIGS. 14A to 14D illustrate an example inwhich the position of the image center (indicated by a broken line) isnot changed, but the other parts are deformed. The reference location isnot limited to the center of an image. For example, the image center maybe defined as an image after the first alignment unit 332 performs ahorizontal direction alignment, or an end of the image may be defined asa reference instead. The references are to be aligned between aplurality of images on which a second alignment is to be performed.FIGS. 14A to 14D illustrate an example in which the retina shape is madeflat with reference to the boundary line, but embodiments of the presentdisclosure are not limited thereto. For example, an approximation with asecondary curve may be performed on a detected retina layer, and theretina layer may be deformed to be flat with reference to theapproximation curve.

Alternatively, instead of making the retina shape to be flat, a rotatingcomponent of the retina layer may only be corrected. FIG. 14Dillustrates an example in which a rotation is corrected. FIG. 14Dillustrates examples of a tomographic image and a horizontal lineprojected image deformed such that the positions in the depth directionof the retina layer of both ends of the image can be in alignment withreference to the image center. For the correction, for example, anarbitrary boundary line or an approximation curve may be used to rotatethe line about the image, and the tomographic image is rotated such thatthe positions in the depth direction of the image both ends (X=0, X=thenumber of A-scans) of the rotated line can be in alignment.Alternatively, correction in the depth direction may be performed oneach A-scan based on a difference value between a rotated line and aline before the rotation.

After the slope of the retina layer is corrected by using the method,the horizontal line projected image averaged in the X-axis direction canbe generated. By using the horizontal line projected image, the retinalayer position in the depth direction can be grasped.

Variation Example 3

According to this embodiment, the third alignment unit 335 performsalignments area by area of the retina in the horizontal direction(x-axis) and the depth direction (z-axis). ROIs for local alignments inthis case are set to superimpose on each other, for example. However,ROIs for local alignments may not necessarily superimpose on each other.For example, in a case where an area of a set ROI has a sufficient imagefeature value, the ROI may be set not to overlap. ROIs to be overlappedand ROIs not to be overlapped may be set in one image simultaneously. Animage feature is an example of a result of an analysis on a tomographicimage and may include a change of shape, instead of horizontally flat ofa blood vessel, a disease, or the retina, for example. A feature of ablood vessel or a disease can be detected from a tomographic image or aline projected image. The retina shape can be grasped by using aboundary line L1 (ILM) or a boundary line L3 (ISOS). It should be notedthat because a plurality of tomographic images are images acquired bycapturing images of an identical location, a feature may not be detectedfrom all of the plurality of tomographic images but may be detected froma representative one image (such as a reference image).

Alternatively, whether an ROI is to be overlapped or not may be changedsimply in accordance imaging angle of view. For example, in imaging withan angle of view as narrow as 3 mm, ROIs may be overlapped because thereis a high possibility that the retina shape is imaged to be flat in awide range of the resulting image. In imaging with an angle of viewimaging as wide as 10 mm, ROIs may be overlapped with a small overlappedarea or may not be overlapped because there is a high possibility thatthe retina shape is imaged to be flat only in a part of the resultingimage.

Not only changes of an overlapped area of ROIs but also an ROI searchrange can be dynamically changed in accordance with an image feature oran angle of view.

Variation Example 4

According to this embodiment, the image composition unit 336 generates amask image for a local alignment in order to remove an invalid area inan averaging process. However, embodiments of the present disclosure arenot limited thereto. An invalid area may be detected from a tomographicimage without generating a mask image, and the invalid area may beexcluded from the averaging calculation.

Variation Example 5

According to this embodiment, an alignment for higher quality OCT imageshas been described. However, embodiments of the present disclosure arenot limited thereto. For example, the present disclosure is applicableto an alignment for OCTA (OCT Angiography). According to OCTA, forvisualization of a blood flow part, an identical part is imaged aplurality of number of times, and alignments are performed on thecaptured images. After that, a change part between the tomographicimages is calculated. A change between tomographic images can becalculated by calculating a decorrelation, for example. By calculatingsuch a decorrelation, a moving part (such as a blood flow) only remainsand may be visualized to generate an OCTA image. The present disclosureis also applicable to a tomographic image alignment for the OCTA imagegeneration.

Variation Example 6

Having described, according to this embodiment, operations from imagingto display are described as a flow, embodiments of the presentdisclosure are not limited thereto. For example, data that have alreadybeen imaged may be used to perform the high-quality image generationprocess. In this case, the processing relating to the imaging may beskipped, but, instead, a plurality of tomographic images that has beenalready imaged may be obtained. Then, the high-quality image generationprocess is performed in step S304.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-172333 filed Sep. 7, 2017, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: a firstalignment unit configured to perform an alignment in a horizontaldirection on a plurality of two-dimensional tomographic images accordingto a first method, the plurality of two-dimensional tomographic imagesbeing obtained based on measurement light controlled to scan anidentical position of an eye; and a second alignment unit configured toperform an alignment in a depth direction on the plurality oftwo-dimensional tomographic images according to a second method that isdifferent from the first method.
 2. The image processing apparatusaccording to claim 1, wherein the first alignment unit performs thealignment in the horizontal direction on the plurality oftwo-dimensional tomographic images based on a positional deviationamount acquired by comparing projected images in the depth directiongenerated from the plurality of two-dimensional tomographic imagesaccording to the first method.
 3. The image processing apparatusaccording to claim 1, wherein the second alignment unit performs thealignment in the depth direction on the plurality of two-dimensionaltomographic images based on a positional deviation amount acquired bycomparing layer boundaries detected from the plurality oftwo-dimensional tomographic images according to the second method. 4.The image processing apparatus according to claim 3, wherein the secondalignment unit performs the alignment in the depth direction on theplurality of two-dimensional tomographic images based on a positionaldeviation amount of the layer boundary acquired from each of a pluralityof regions dividing the two-dimensional tomographic image in thehorizontal direction according to the second method.
 5. The imageprocessing apparatus according to claim 4, wherein the second alignmentunit selects a predetermined number of positional deviation amounts frompositional deviation amounts of a plurality of the layer boundaries inthe plurality of regions such that a value indicating a variation of thepositional deviation amount in the predetermined number of combinationsis at a minimum, and performs the alignment in the depth direction onthe plurality of two-dimensional tomographic images based on arepresentative value of the selected positional deviation amounts,according to the second method.
 6. The image processing apparatusaccording to claim 5, wherein the value indicating a variation includesa variance or a standard deviation; and wherein the representative valueincludes an average value or a median value.
 7. The image processingapparatus according to claim 1, further comprising: an image selectionunit configured to select a reference image being a reference for analignment from the plurality of two-dimensional tomographic images,wherein the image selection unit selects the reference image based on atleast one of a result of a comparison between projected images in thedepth direction acquired from the plurality of two-dimensionaltomographic images and a result of a comparison between layer boundariesacquired from the plurality of two-dimensional tomographic images. 8.The image processing apparatus according to claim 7, wherein the resultof the comparison between projected images in the depth directionacquired from the plurality of two-dimensional tomographic imagesincludes a similarity between the projected images; and wherein theresult of the comparison between layer boundaries acquired from theplurality of two-dimensional tomographic images includes a positionaldeviation amount of the layer boundaries.
 9. The image processingapparatus according to claim 8, wherein the image selection unit selectsthe reference image based on, in a case where one two-dimensionaltomographic image and a plurality of other two-dimensional tomographicimages are compared in the plurality of two-dimensional tomographicimages, one of a number of two-dimensional tomographic images having apredetermined threshold or higher value of the similarity and having apredetermined threshold or lower positional deviation amounts of thelayer boundaries and a number of two-dimensional tomographic imageshaving a predetermined or higher number of regions dividing each of thetwo-dimensional tomographic images in the horizontal direction, theregions having a predetermined threshold or lower positional deviationamounts of the layer boundaries.
 10. The image processing apparatusaccording to claim 7, further comprising: a third alignment unitconfigured to perform an alignment on the plurality of two-dimensionaltomographic images having undergone the alignment by the first alignmentunit and the second alignment unit with reference to the referenceimage; and an image selection unit configured to select a referenceimage for the alignment to be performed by the third alignment unit fromthe plurality of two-dimensional tomographic images.
 11. The imageprocessing apparatus according to claim 10, further comprising ageneration unit configured to generate an averaged image by averagingthe plurality of two-dimensional tomographic images having undergone analignment by the third alignment unit.
 12. The image processingapparatus according to claim 10, wherein an OCTA image is generated fromthe plurality of two-dimensional tomographic images having undergone analignment by the third alignment unit.
 13. An alignment methodcomprising: performing an alignment in a horizontal direction on aplurality of two-dimensional tomographic images according to a firstmethod, the plurality of two-dimensional tomographic images beingobtained based on measurement light controlled to scan an identicalposition of an eye; and performing an alignment in a depth direction onthe plurality of two-dimensional tomographic images according to asecond method that is different from the first method.
 14. Anon-transitory computer-readable storage medium storing a program forcausing a computer to execute the method according to claim 13.